1. Field of Invention
The present invention relates to an image pickup device capable of changing, while the photographing is under way, the repetition period of a synchronization signal (a frame rate), which provides a timing reference for the frame-based operation of image pickup device.
Priority is claimed on Japanese Patent Application No. 2007-102492 filed on Apr. 10, 2007, the content of which is incorporated herein by reference.
2. Description of Related Art
X-Y address type image pickup elements, typically exemplified by a CMOS image sensor, which can be readily controlled for charge storage and readout of photographed image signal on a row-by-row or column-by-column basis, are favored because of their capability of performing high speed imaging at low power consumption. Those X-Y address type image pickup elements are likely to find increasingly broader use in view of their high speed, low power performance. On the other hand, an X-Y address type image pickup element inherently characterized by the rolling readout of image signal data on a row-by-row or a column-by-column basis, under which the sequential charge storage (rolling shutter) of image signal data is performed for those photographed image signals with readout timing virtually coincident with the start/end of the charge storage, has a disadvantage of causing in-frame deformation of a moving image due to the row-to-row or column-to-column storage time lag (to be described in more detail below). While a more advanced X-Y address type image pickup element has been developed in recent years, which is capable of simultaneous storage (global shutter), such an image pickup element still has those various problems of diminished performance in terms of sensitivity, saturation and noise, and of tendency to cause blooming, which are attributed to element structure involving a greater number of transistors provided for each pixel. These are the problems to be solved before such a recently developed image pickup element can be put into practical use.
Japanese Patent Application, First Publication No. H7-298112 (referred to as Reference 1 hereunder) teaches an image pickup device capable of arbitrarily changing the frame rate of the image signal by changing the repetition period of the synchronization signal.
Japanese Patent Application, First Publication No. H7-50786 (referred to as Reference 2 hereunder) shows another image pickup device capable of expanding dynamic range while reducing noise, by the operating process of combining a plurality of consecutive field signals into one frame signal.
In an X-Y address type image pickup element, a change in both the storage/readout period and the frame rate entails the problems outlined below. It is assumed here that the X-Y address type image pickup element is of the type capable of row-by-row rolling.
First Problem
Referring to FIG. 13 illustrating an imaging operation performed at an ordinary frame rate, it is assumed that the photographed image 1300 of an object moves horizontally within one frame period along the major plane of the image pickup element. The part denoted by symbol (a) in FIG. 14 shows a one-frame portion of the photographed image, while the part denoted by symbol (b) in FIG. 14 shows the timing for the operation of the image pickup element. Image signal data is read out within one frame period, which is defined by synchronization signal 1400, from the first row to the last row of pixels constituting the image pickup element. In the part (b) of FIG. 14, a point 1410 denotes a readout start timing for the first row of the image sensor, while a point 1420 denotes a readout start timing for the last row of the image sensor.
The part denoted by symbol (a) in FIG. 15 shows a row-by-row illustration of pixels constituting the image pickup element, while the part denoted by symbol (b) in FIG. 15 shows timing for the operation of the image pickup element in greater detail than the part (b) of FIG. 14. Image signal data is read out from each of the rows during one scanning line period, which is defined by the repetition period of synchronization signal 1500. More specifically, taking the first row for example, at the timing of the completion of the charge storage of the immediately preceding frame, the stored video signal data is read out simultaneously from all the pixels in the first row, and then temporarily stored in a signal holding member (not shown) consisting of capacitors and the like, allowing the signals from each of the pixels to be sequentially read out in a single scanning line period. The signal data storage for the next frame is started virtually at the same timing as the completion of the readout from the current frame of the image signal.
Since the read out timing for the first scanning line and that for the last scanning line are not identical as described above, the photographed image suffers the so-called moving image deformation as shown by horizontally deformed photographed image 1430 in the part (a) of FIG. 14. In another image pickup element adapted to perform column-by-column rolling operation, the deformation direction shown in the part (a) of FIG. 14 will be reversed in terms of horizontal/vertical directions of the display.
Description will now be given concerning the situation where the imaging is performed at a low frame rate. Referring to FIG. 16, it is assumed that photographed image 1600 moves horizontally along the major plane of the image pickup element within one frame period as in the above-described case of performing the imaging at an ordinary frame rate. The part denoted by symbol (a) in FIG. 17 shows the image of one frame which was imaged at this time while the part denoted by symbol (b) in FIG. 17 shows the timing for the operation of the image pickup element. As shown in the part (b) of FIG. 17, the low frame rate imaging, under which the frame period is longer than the ordinary frame rate imaging, allows the object to be photographed to move over a greater distance within a frame period than in the ordinary frame rate imaging. In addition, there is a greater time lag between the readout of the first scanning line and that of the last scanning line. As a result, the horizontal deformation becomes greater as shown by photographed image 1700 in the part (a) of FIG. 17.
As described above, an image pickup element adapted to perform the rolling operation involves the problem of the moving image deformation of photographed object, which becomes particularly conspicuous for low frame rate imaging. However, the above reference 1 shows nothing about countermeasures to the problem described above.
Second Problem
In some cases, a photographer may choose a mode of operation which allows the frame rate and the storage time for the photographed image to be gradually changed, depending on the motion of the object to be photographed. Such mode of operation is frequently chosen to achieve special effects, since it allows a slow replay of a moving image recorded at a higher frame rate or a fast replay of a moving image recorded at a lower frame rate. In the case of the gradual change of frame rate, the storage time is changed along with the change in frame rate to compensate for the out-of-focus state caused by the motion, thereby to produce more natural slow motion or fast motion moving images
An image pickup element adapted to perform the rolling operation, in which the end of storage operation and the start of readout operation are brought into coincidence, involves constraints in its operation outlined below. In general, the operation of an image pickup element is controlled by an electrical pulse on a row-by-row or column-by-column basis. In an X-Y address type image pickup element adapted to the rolling operation, the pulse is provided by a simple circuit structure, wherein the supply of a single original pulse is delayed through a shift register on the row-by-row or column-by-column basis. It is therefore impossible to simultaneously perform different controls concerning a plurality of rows or columns within a frame. More specifically, during the charge storage/readout control performed for the current frame, the same control for the immediately following frame cannot be performed.
Further description will now be given assuming that the frame rate is fixed while the charge storage time (storage time) is changed. In FIG. 18, a frame rate defined by the repetition period of synchronization signal 1800 is fixed and the changeable storage time controlled by the charge storage start timing control is performed by the charge resetting. In FIG. 18, the charge storage control for the next frame is performed during periods A, B and C, in the midst of the on-going charge storage control for the current frame. More specifically, during period A, for example, prior to the start of the charge storage for the last row of the current frame, the charge storage for the first row of the next frame is started. To achieve the performance described above, a plurality of control circuits, for example, need to be provided for the complicated control. Therefore, it is impossible for the above-described arrangement to cope with the situation, where the charge storage time is to be changed greatly and frequently for consecutive frames so that both frame rate and charge storage time may be changed greatly to achieve special effects for moving images.
Third Problem
Some of the recently developed digital still cameras and/or monitoring cameras are adapted to combine image signals obtained from two or more consecutive frame periods into a frame signal of one frame length. This type of cameras suffer conspicuous motion-induced blur in the photographed image of the synthesized frame signal, when the object to be photographed moves very fast, because the position, on the major plane of the image pickup element, of the photographed image shifts from one frame to another. Particularly, if the frame rate is changed at random, the degree of the motion-induced blur varies at random, resulting in extremely unnatural reproduced image, because the system of Reference 2 combines the image signals from a plurality of frames with fixed processing parameters. In addition, since noise generated at the image pickup element varies depending on the charge storage time, noise component appearing in the reproduced moving image is changed at random, making quite unnatural the moving image viewed at the time of replay, if the image signal is obtained from a plurality of frames with the charge storage time changed at random.
Fourth Problem
Change of the frame rate and the charge storage time to achieve a desired special effect can bring cause a vacant period between every two neighboring frame periods, where neither the charge storage nor the readout is performed. Such vacant periods can cause intermittent motion in the image of a moving object to be photographed, resulting in a very unnatural reproduced image.
In view of the First Problem 1 set forth above, it is a first object of the invention to provide an image pickup device capable of reducing the deformation in the reproduced image of an object to be photographed. Also, in view of Problem 2 above, it is a second object of the invention to provide an image pickup device capable of preventing the occurrence of the periods, during which neither the charge storage nor the readout of the image signal data can be controlled. Moreover, in view of Problems 3 and 4 above, a third object of the invention is to provide an image pickup device capable of providing high-quality reproduced images.